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GENESYS DESCRIPTION 



GENESYS GROUPS 



GENESYS is a GENeric Evaluation SYStem. 

GENESYS is a method of streamlining data collection 
and evaluation through use of computer technology. 
Prom year one in 1973, the Office of Research and 
Evaluation (ORE) has been challenged to evaluate a 
multitude of contrasting programs with limited 
resources. By standardizing methods and information 
provided* GENESYS makes it possible to evaluate a 
much larger number and variety of programs than 
would ordinarily be possible. GENESYS gathers and 
reports the following standard information on specified 
groups of students: 



Student characteristics 

Achievement 

Attendance 

Discipline 

Grades/credits 

Dropouts 

Retainees 



GENESYS can be run for any group of students 
identifiable through a computer file. Most of the 
groups included this second year were for students 
served in 1989-90; some were followups of groups 
served in 1987-88. A complete listing is shown in the 
right-hand column of this page. References to other 
publications and reports which incorporate GENESYS 

data are provided in this paper. 



GENESYS included a wide variety of elementary, 
secondary, and K-12 programs in its second year. 
Students were served in 1989-90 unless otherwise 
noted. 

K-12 

Bilingual/ESL 

PAL 

CIS 

Project Mentor 

Elementary 

Teach and Reach 
AIM High 
DARE, 1987-88 

Secondary 

Libera] Arts Academy 

Healing Magnet 

Secondary Honors Program 

Science Academy — NSF Grant 

Sixth Graders— 1989-90, 1988-89, 1987-88 

TAP 

AIP 

Title VII 
Project GRAD 
CVAE 
PEAK 

Alternative Learning Center 
Zenith 

Johnston Computer Lab 
Dropouts 
Evening School 
Teenage Parent Program 
Johnston Dropout Recovery 
Crockett Project Touch 
Martin Initiative 
Academic Decathlon 
Johnston Renaissance 
Robbins Secondary School 



Nate to Readers: The authors realize these program names 
and acronyms are too cryptic for most readers. The list is 
included to illustrate the number and range of programs 
evaluated. 
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The idea of a generic evaluation system has been 
conceptualized and reconceptualized for years* In 
1989, the shrinkage of staff resources, the growth 
in information needs, and improvements in 
technical capabilities combined to permit the 
creation of GENESYS in concrete form. The 1989- 
90 school year is the second year of GENESYS 
implementation* Readers interested in more 
information about the development and imple- 
mentation of GENESYS in its first year, 1988-89, 
are urged to consult the reports listed in the 
reference section. 

WHAT IS GENESYS? WHY IS IT NEEDED? 

GENESYS is ORE's GENeric Evaluation SYS tern. 
Broadly speaking, GENESYS is: 

• A method of streamlining data collection 
and evaluation for a wide variety of 
projects, 

• A means to gather and report a great deal 
of information on the characteristics and 
outcomes for particular groups of students, 

• A mechanism to evaluate a multitude of 
contrasting programs with limited re- 
sources—especially limited time, 

• A way to provide valuable outcome 
information on more programs than would 
ordinarily be possible given limited 
evaluation resources, 

• A method for responding to the challenge 
of requests for last-minute, instant 
program evaluation information, 

• A way that program staff, administrators, 
and members of the Board of Trustees can 
obtain information on the progress of 
students involved in particular programs 
or innovations which would otherwise be 
unavailable because of scant evaluation 
resources, 

• A way that evaluation staff for various 
projects can obtain standard information 
for various programs, thus allowing 
comparisons across projects as well as 
freeing up staff time to do more sophisti- 
cated analyses for areas not covered 
sufficiently by GENESYS, and 



• A means to uncover trends or interesting 
findings on projects that bear delving into 
more thoroughly. 

Specifically, GENESYS is: 

• A data-base methodology accessing the 
school system's available longitudinal data 
bases, and 

• A set of computer programs utilizing the 
Statistical Analysis System (SAS) which 
have been written and linked to generate 
standard output on a number of variables 
for designated programs. 

One limitation of GENESYS is that it may not 
provide everything a user wants in the exact form 
desired* It also reports the same information for 
each program. Users must exercise their own 
judgment about which variables are the best 
measures of success for their program* On the 
other hand, the standard format and definitions of 
GENESYS have facilitated comparisons across 
programs. Other limitations of using GENESYS 
are elaborated in full in two ORE publications, 
88.40 and 88.36 (see reference list). 



HOW DOES GENESYS WORK? WHAT DOES 
GENESYS PROVIDE? 

Given a file of the student identification numbers 
of those students involved in a program, group, or 
innovation, GENESYS will provide outcome 
information for the following variables: 

GROUP CHARACTERISTICS: Number served 
by grade, ethnicity, sex, low income, LEP, 
overage for grade, special education, 
gifted and talented; 

1989-90 ACHIEVEMENT RESULTS BY 

GRADE: ITBS, TAP, TEAMS and 1988-89 
to 1989-90 ITB&TAP regression trend 
information; 

ATTENDANCE, DISCIPLINE, GRADES/ 
CREDITS: 1988-89 and 1989-90 (four 
semesters); and 

DROPOUTS AND RETAINEES: Dropouts as 
of the end of the fifth six weeks and 
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potential retainees as of the end of May, 
1990 (actual retainees and dropouts as of 
the end of the 1989-90 school year, 
updated in fall, 1990). 

Specific definitions for each of these variables are 
included in Attachment 1. The user is advised to 
read and refer to the definitions provided to 
assure correct interpretation of the data. 

For each group, three types of sheets are pro- 
duced, 

THE GENESYS EVALUATION SUMMARY 
summarizes information on the group's 
overall performance on all variables. 

THE EXECUTIVE SUMMARY summarizes 
findings in more narrative form and 
compares the program's data to relevant 
comparison groups. On most variables, 
comparison is to the AISO average for the 
appropriate grade span — AISD elemen- 
tary, middle/junior high, or senior high 
students. Attachment 1 provides addi- 
tional information about GENESYS 
comparisons. 

GENESYS DATA BY STUDENT provides a 
listing of this information by student (as 
applicable) to allow a specific review of 
student attainment and characteristics 
(Attachment 2). 

A brief program description is also supplied by 
program or evaluation staff. The sections which 
follow show sample program descriptions, and 
evaluation and executive summaries. 

Two optional printouts were added to GENESYS 
in 1989-90. 

CROSS-PROGRAM COMPARISON CHARTS 
provide a summary of statistics across 
multiple programs designated by the user. 

TWO-WAY CROSSTABULATION TABLES 
provide a greater level of detail about 
selected variables than that provided in 
the evaluation summary. 



Attachment 3 is a sample GENESYS report for 
Austin s Liberal Arts Academy. 



WHAT IS NEEDED TO RUN GENESYS? 

GENESYS needs a file of student identification 
numbers for the program or group which is to be 
studied before it can be run. Gathering this 
information is the responsibility of the program or 
evaluation staff requesting the information. 
Student names and identification numbers can be 
provided as a list, on a computer disk, or as a 
description of critical location information on 
AISD computer files (such as a school and grade 
list or a course number). Staff must decide 
whether they want to include all students served 
for any length of time by a program, those in as of 
a particular date, or those served a certain length 
of time (e.g., over three months). This choice 
should be communicated to ORE with the list. In 
addition, staff are asked to provide a brief pro- 
gram description. 

Generally, GENESYS can be run at any time after 
first semester records are in for the current year. 
Of course, information is available for more 
variables and is more complete at year's end. 
GENESYS can also be run based on the previous 
year's data. Attachment 4 provides flow charts for 
GENESYS. 



WHAT PROGRAMS ARE INCLUDED IN 
GENESYS? 

A list of programs and groups included in 
GENESYS in 1989-90 is shown in Figure 1. As of 
June, 1990, 56 groups have been run through 
GENESYS this spring. The first groups listed are 
included in this report because they are not 
discussed in other ORE reports. They should 
provide a good sampler of what GENESYS is all 
about to the reader. Results for the rest are 
included in the other ORE reports referenced. A 
complete set of results for other groups of interest 
is available upon request from ORE. 
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FIGURE 1: GENESYS GROUPS— 1989-90 



PROGRAM/GROUP 
Keating Magnet 

Johnston Liberal Arts Academy 

Teach and Reach, 1989-90 

AIM High (Gifted/Talented Program 

Secondary Honors Program 

Bi lingua 1/ESL Programs 

LBJ Science Academy 

Sixth Graders, 89 90, 88^89, 87-88 

Academic Decathlon 

Drug Abuse Resistance 
Education (DARE). 1987^8 

Title VII 

Project GRAD 

Academic Incentive Program ( AIP), 
1989*90 

Alternative Learning Center ( ALC) 

Communities In Schools (CIS) 

Coordinated Vocational Academic 
Education <CVAE) 

Crockett Project Touch 

Evening School 

Johnston Computer Lab 

Johnston Renaissance 

Johnston Dropout Recovery 

Martin Hispanic Student Scholarship 
Initiative 

Peer Assistance and Leadership 
(PAL) 

Practical, Effective, Appropriate 
Knowledge (PEAK) 



PUBLICATION # 
89.30 



REPORT TITLE 

GENESYS 1989-90: Selected Program Evaluations 

GENESYS 1389-90: Selected Program Evaluations 89.30 

GENESYS 1989-90: Selected Progmm Evaluations 89.30 

GENESYS 1989-90: Selected Program Evaluations 89.30 

GENESYS 1989-90: Selected Program Evaluations 89.30 

GENESYS 1989-90: Selected Program Evaluations 89.30 

Double TNT: Targeting New Teachers and Teaching by Novel Techniques 89.27 

Sixth Graders in Elementary and Middle Schools: A Longitudinal Comparison 89.31 

Chapter 2 Formula, 1989-90: Major Points 89.32 

Continued Steps Toward Drug Free Schools in AISD. 1989 90 89.38 

Title VII in AISD, 1989-90 89.39 

Continuing Initiatives in Dropout Prevention: Project GRAD Final Report, 89-90 89.35 

Continuing Initiatives in Dropout Prevention: Project GRAD Final Report, 89-90 89.35 

Continuing Initiatives in Dropout Prevention: Project GRAD Final Report, 89-90 89.35 

Continuing Initiatives in Dropout Prevention: Project GRAD Final Report, 89-90 89.35 

Continuing Initiatives in Dropout Prevention: Project GRAD Final Report, 89-90 89.35 

Continuing Initiatives in Dropout Prevention: Project GRAD Final Report, 89-90 89.35 

Continuing Initiatives in Dropout Prevention: Project GRAD Final Report, 89-90 89.35 

Continuing Initiatives in Dropout Prevention: Project GRAD Final Report, 89 90, 89.35 

and Chapter 2 Formula, 1989-90. Major Points 89.32 

Continuing Initiatives in Dropout Prevention: Project GRAD Final Report, 89 90 89.35 

Continuing Initiatives in Dropout Prevention: Project GRAD Final Report. 89-90 89.35 

Continuing Initiatives in Dropout Prevention: Project GRAD Final Report, 89-90 89.35 



Continuing Initiatives in Dropout Prevention: Project GRAD Final Report, 89-90. 89.35 
and Continued Steps Toward Drug-Free Schools in AISD, 1989-90 89.38 

Continuing Initiatives in Dropout Prevention: Project GRAD Final Rrport t 89 90 89 35 



Project Mentor 


Continuing Initiatives in 


Dropout Prevention: 


Project GRAD Final Report, 89 90 


89.35 


Rohbins Secondary School 


Continuing Initiative* in 


Dropout Prevention: 


Project GRAD Final Report. 89-90 


89.35 


Teenage Parent Program 


Continuing Initiatives in 


Dropout Prevention: 


Project GRAD Final Report, 89 90 


89.35 


Transitional Academic Program 
(TAP). 198990 


Continuing Initiatives in 


Dropout Prevention: 


Project GRAD Final Report, 89 90 


89 35 


Zenith Program 


Continuing Initiatives in 


Dropout Prevention: Project GRAD Final Report, 89 90 


89 35 
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WHAT ENHANCEMENTS HAVE BEEN MADE 
TO GENESYS IN 1989-90? 

Some of the enhancements the evaluation staff 
who developed GENESYS hoped to make in 1989- 
90 have been realized, while other ideas are stilt 
on the drawing board. Some promising new ideas 
have emerged for future development The 
following is a list of the improvements and en- 
hancements made to GENESYS this year. 

• The Evaluation Summary, formerly the 
Program Summary, wm redesigned to be 
easier to understand and use as well as be 
more attractive. 

• An additional retainee variable wag abided 
to the evaluation summary, and the 
previous variable was renamed. The 
variable "retained,* defined as the percent- 
age of students recommended for retention 
as of May, now refers to "end-of-year" 
retainees. A "beginning-of-year* variable, 
defined as the percentage of students 
actually retained as of the beginning of the 
next school year, was added* 

• The Executive Summary was rewritten to 
make it less narrative and more a graphi- 
cal display of data. 

• Results from the evaluation summary 
were saved on a disk file for the first time. 
The evaluation summary for a group can 
now be recreated, even modified (e.g., if 
the title needed to be changed), without 
running the group through all of the 
GENESYS programs again, thus saving 
considerable computer time. 

• The percentage of students who are gifted/ 
tateatefl added to the evaluation 

su ninary. 

• Thf heaiflffig fpr foe Data bv Student 
listing was redesigned to be printed in 
reverse-font by the laser printer. 



Some additional standardization efforts were 
made. 

• A file/run sheet was devised for the benefit 
of users. This sheet provides users with a 
kind of checklist to help them work 
through some of tho issues involved in file 
building. It also assists the programmer in 
running the group. Finally, it serves as 
valuable documentation of how the file was 
assembled, especially as regards what 
students were included in a group. 

• Users were given more precise instructions 
on how to prepare the input files for their 
groups. They were directed to eliminate 
bad and duplicate student ID numbers 
from their data files and were provided 
with a SAS program for the purpose, 

• "gpanried* gro u ps. i.e., groups in which 
there were students in different grade 
spans such as middle/junior high school 
and high school, were not permitted. 
Groups had to be defined as either elemen* 
tary, middle/junior high school, or high 
school. 

• Group size was limited to a minimum of 25 
students both in the interest of meaningful 
analysis and to save computer run time. 

Two user-designated options, to be run apart from 
the main GENESYS processing, were made 
available. 

• Cross-program comparison charts compare 
statistics across programs selected by the 
user. A minimum of two programs ran be 
designated, up to the maximum of all the 
programs run. If cross-program compari- 
sons are specified, the user receives all of 
the charts; i.e., it is not an option to choose 
only certain comparisons* Programs are 
compared on all GENESYS demographic, 
progress, and achievement indicators. A 
complete set of comparison charts for fall, 
1989, programs is contained in ORE 
publication 89.30 (see reference list). 
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• Two-way crosstabulabon tables (e,g. ? sex 
by ethnicity) permit the user to examine 
program data at a greater level of detail 
than tliat presented in the GENESYS 
evaluation summary. The user is able to 
select certain "Mocks* of categorical 
variables for which all possible two-way 
tables will be printed For example, a user 
may be interested in a crosstabulation of 
sex by grade for a particular group of 
students. In addition to this table, the 
user would receive crosstabulations of 
grade by all other categorical variables* 
Crosstabulations by continuous variables, 
e.g., of percent attendance, are not pres- 
ently included. A list of the tables in- 
cluded in each block is also included in 
ORE publication 89.30. 

WHAT CHALLENGES REMAIN, AND WHAT IS 
PLANNED FOR THE FUTURE? 

Although a number of enhancements were made 
to GENESYS in 1989-90, there is still room for 
improvement GENESYS remains a complicated 
development and production process which 
requires considerable time and attention from 
evaluation staff to do the programming, coordina- 
tion, and set-up work. 

Developing Program Files and Descriptions 

One facet of the process which took longer than 
expected in the first year of GENESYS, 1988-89, 
was the development of program files and descrip- 
tions. Slowdowns were attributed generally to the 
following factors: 

• Deciding which students should be 
included in data files, 

• Deciding what sources should be used for 
files, 

• Difficulty in collecting basic program 
information, and 

• Difficulty in collecting students* names 
and ID numbers when they are not already 
computerized. 

These difficulties remained in 1989-90, although 
some attempts have been made to delineate the 



issues— starting with the 1988-89 GENESYS final 
report — and to arrive at a common frame of 
reference. Attachment 5 f "Requirements for 
GENESYS Data Files," which was distributed to 
GENESYS users in spring, 1990, was one at- 
tempt Another was the development of the file/ 
run sheet which was described in the previous 
section. Some discussion with the evaluation staff 
responsible for GENESYS helped to clarify 
questions about who should be include? »n data 
files. 

A second year's experience with the programs on 
the part of evaluation staff helped them in making 
decisions about programs with which they were 
not as familiar last year. Where program staff 
had concerns last year about the criteria used for 
inclusion in a group, evaluation staff were able to 
address them more readily because of their 
greater familiarity with the programs and with 
the GENESYS structure. 

Some of the demands on staff resources will lessen 
as staff acquire additional experience and the 
process becomes more routine. However, some of 
these demands may be irreducible parts of the 
"business* of evaluation. Just as it is an ongoing 
part of Data Services to work with users to deter- 
mine how best to meet thtir needs, so too may 
evaluation staff have to continue to work with 
GENESYS users to educate them and to ensure 
that the information they are seeking can be 
provided most efficiently via GENESYS. 

Additional Challenges for the Future 

Even at the end of the second year of implementa- 
tion, the system is still less "user friendly" thqp 
desired Nonprogrammer users still cannot 
submit their own runs. Other computer program- 
mers could run GENESYS, but because the 
system has kept changing and evolving, it seemed 
risky to the evaluation staff responsible for 
GENESYS to let anyone besides the main 
GENESYS programmer handle GENESYS runs. 
As the system becomes more stable and better 
understood both in ORE and outside of it, it will 
be possible to permit users greater, less encum- 
bered access to GENESYS. 

A related use issue is that few people outside of 
ORE are directly invnlvgd in u&in ? RRNESVR 
There are many recipients of GENESYS informa- 
tion, but few people have requested that 
GENESYS be run on groups of interest to them. 

*> ~ 



90.21 

Lower Costs, Fewer Staff, More Information: Critique of a Generic, Data-Base Evaluation System 



This lack of direct involvement is probably attrib- 
utable to the relative newness of GENESYS. 
District staff have indicated a general awareness 
of GENESYS but not a thorough understanding of 
what information it can provide. Another plau- 
sible explanation is that ORE's current broad 
inclusion of programs has left few others of 
interest. 

dealing pr ogram descriptions is still not as "push 
hutW as desirable for a generic evaluation 
system. Program descriptions are supplied by 
program or evaluation staff, but evaluation staff 
ensure that the descriptions are accurate and are 
typed on the standard form. This process is still a 
paper-and-penctl affair. One possibility for 
improving this process next year is to set up a 
central computer file on the mainframe into which 
program descriptions would be typed. The file 
could be accessed through any terminal in ORE, 
Program descriptions would be saved and could be 
altered at any time. When GENESYS output is 
created for a group, program descriptions could be 
laser printed at the same time as the summaries 
and individual student listings. 

Running GENESYS in bnth fall and spring has 
addod tn the time invited in the system and led 
to questions about what groups should be run 
when. In its first year, 1988-89, a limited number 
of fall runs were made to test computer programs. 
In 1989*90, however, 38 programs were run in the 
fall and 56 in the spring (through June). This 
represents a substantial commitment in computer 
time, as well as in staff time. After only two 
years, it is evident that GENESYS has become a 
mfigor evaluation tool, so much so that a more 
judicious selection of groups to be run may be 
necessary, at least given the present capabilities 
of the system. 

In light of the potential and growing demand for 
GENESYS information, GENESYS run time 
needs to he reduced. Even with a faster IBM 
mainframe than ever before, it takes 20-30 min- 
utes to process the GENESYS computations for 
one program group. What this means, with 
upwards of 100 groups (many after June) pro- 
cessed in spring, 1990, is that a substantial 
amount of computer time is being devoted to 
GENESYS. At the rate of about five groups a 
night, the large number of groups and the long 
run time mean that the programmer is running 
GENESYS every weekday night for a month and 
longer. One possibility which has been discussed 



is to rewrite parts of the GENESYS computer 
programs in COBOL rather than SAS. COBOL is 
better suited for extracting information from large 
files, while SAS is superior for manipulating the 
data and producing statistical output. 

Some additional enhancements to GENESYS are 
being considered. Attachment 6 lists some ideas 
for enhancements broached in spring, 1990, some 
of which have already been implemented. Two of 
these ideas in particular merit some discussion 
here: 

1 . Comparison of predicted and obtained 
dropout rates, and 

2. Significance testa 

The comparison of predicted and obtained dropout 
rates is an outgrowth of some work done in 1988- 
89 as part of the evaluation of the District's 
dropout prevention programs. The 1988-89 
Project GRAD final report (Publication No. 88.36) 
includes a discussion of how the rates are ob- 
tained and compared (see pages IV-32 - IV-35). 
This methodology has been incorporated into 
GENESYS to provide another outcome indicator 
which is more than descriptive. A more complete 
discussion of this methodology is contained in 
ORE publication 90.19. 

Significance tests for GENESYS are an exciting 
concept because they would provide an additional 
evaluative dimension not now furnished by 
GENESYS, namely, a means for determining if 
the differences between groups (either between 
program students and students district wide or 
program students at two points in time) are 
meaningful. Several avenues for introducing 
significance tests are being investigated, 

WHAT MAKES A PROGRAM EVALUABLE BY 
GENESYS? 

After two years of interpreting the output from 
GENESYS, we have determined that indeed there 
are certain characteristics that make a program a 
poor candidate for GENESYS — or at least parts of 
it. The m^jor characteristic appears to be the 
nature of the enrollment of the students. 

Some programs, such as our alternative campus 
for discipline problems, can enroll students at any 
time during a semester This make it impossible 
to attribute the influence of the time in the 
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program and the time in regular education to the 
outcomes. For example, how much of a student's 
GPA would be attributable to each program if 
time were split during a semester? The same 
difficulty arises with programs that are less than 
a calendar year when interpreting annual out- 
come measures such as achievement test scores, 
dropout rates, or promotion/retention rates. 
Therefore, one of the most critical characteristics 
of an ideal program for a GENESYS evaluation is 
that the students begin the program and continue 
in the program for the full duration of the interval 
between the measurement of outcome variables. 

An option is to specify for inclusion in GENESYS 
only those students who are enrolled for the full 
duration. However, this could reduce the number 
of students contributing to the evaluation to an 
unacceptable level. 

A second key characteristic is that the program 
have as its targeted outcomes those that are 
reported by GENESYS. For instance, a program 
that targets student self-concept, parental in- 
volvement, or physical development would not be 
assessed directly by any of the measures currently 
included in GENESYS. A third key characteristic 
is that the program not enroll students who are 
exempt from the outcome measures* For example, 
a program that enrolls certain limited-English* 
proficient or special education students might 
have too few of those students appearing in the 
GENESYS statistics by virtue of their exemption 
from many tests and other assessments. 

Two other characteristics are less important. The 
program should have sufficient enrollment to 
make the GENESYS averages meaningful, and 
the program should be able to provide a roster of 
those students efficiently. A m^jor hassle with 
the inclusion of programs in GENESYS is the 
difficulty of obtaining a roster of the program 
participants. Many programs can identify their 
students by course numbers or a combination of 
school/class/period codes. Those programs and 
those that serve all students with certain demo- 
graphic characteristics are usually easy to define. 

To summarize, the ideal GENESYS evaluation is 
one for a program that: 

• Enrolls students for an entire school year 
and ha$ a small attrition rate. 

♦ Targets outcomes that are among those 



included in GENESYS. 

* Serves students that are not exempt for 
those outcome measures. 

* Enrolls enough students to make averages 
meaningful. 

* Serves students who can easily be identi- 
fied through already existing computer 
records or characteristics. 



WHAT DOES GENESYS COST COMPARED TO 
TRADITIONAL EVALUATIONS? 

The best example of a use for GENESYS is the 
actual case study that finally motivated us to 
develop the system. That episode began when one 
of the District's high schools implemented a 
computer lab for potential dropouts. The lab was 
pieced together the way many are around the 
country with a donation/loan from a m^jor com- 
puter manufacturer, curriculum from a prominent 
educational computer software company, and 
prompting from a community entity with enough 
political clout to ensure that the District would try 
out the idea. Everything went along relatively 
unnoticed during the first year until the principal 
of the school was invited to speak at a national 
meeting sponsored by the computer corporation, 
which had made an excellent videotaped docu- 
mentary op some of the students in the program. 

As happens* the national attention brought out 
some of the local ill feelings about the program 
and how it had been approved. Local secondary 
administrators challenged not only the claims of 
success made in the videotape, but also the overall 
success of the community entity, a GED prepara- 
tion agency, which had been quite frankly giving 
the District fits about the number of dropouts 
leaving our high schools as a way of drumming up 
support for its alternative training for a GED. 

Our Office of Research and Evaluation was 
admittedly skeptical about the claims, which 
included the statistic that only 3 of the 40 stu- 
dents enrolled had dropped out. That would have 
been a better rate than the 5 out of 40 dropout 
rate for all students in the District during a single 
school year. 
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After a soul-searching meeting with the Superin- 
tendent and all staff involved, the Office of Re- 
search and Evaluation was asked to investigate 
two issues. 

J. What was the dropout rate of the 40 
students who participated in the pro- 
gram, and what do other measures of 
success indicate? 

2. What documented success have the 

computer curriculum and the community 
entity had with the persons they have 
served? 

In the next three days, about 12 staff in the Office 
of Research and Evaluation contributed nearly 
full time and in some cases more than full time to 
compiling and analyzing available data. A 
spreadsheet format was designed to display the 
records of individual students, because it was 
clear there would be no single measure of success 
that would be acceptable to everyone as the 
bottom line for determination of success. Attach- 
ment 7 is a copy of one page of the final spread- 
sheet that was produced. Because of the quick 
turnaround demanded, the final copy was hand 
written* 

Evaluation staff used available printed reports; 
individual student records kept by project staff; 
computerized files for grades, discipline, atten- 
dance, demographics, and test scores; and SAS for 
analyses. 

The outcome was a report in memorandum form 
that concluded that 12, not 3 students met the 
definition of a dropout; participating students had 
a lower rate of attendance during the program 
than before; participating students raised their 
grade point averages from just below failing to 
just above failing during the fall, but earned 
failing grade point averages below those they had 
earned before the program in the spring; credits 
earned changed little; discipline incidents changed 
little. 

As for the outside entities' program, they were 
unable to provide any published report or other 
documentation of their students' success. Less 
formal data they provided showed that they had 
served many students, of whom most had not 
received a GED. 



Possibly the most important outcome of this 
enormous human effort to assemble an evaluation 
report over the span of only three days was the 
motivation for our office to pull off the shelf all of 
our previous thoughts and plans for a generic 
evaluation system and actually commit ourselves 
to making it work before this situation that we 
had lived through with the computer lab was 
repeated with another program. 

The cost? This computer lab effort probably cost 
about $4500 in staff time over the three days. 
Never before and never since have we pulled so 
much in human resources off regular activities for 
such a quick, ad hoc project. 



HOW MUCH DOES GENESYS SAVE US IN A 
YEAR? 

This is difficult to conceptualize, partly because 
the District would not evaluate all these programs 
without GENESYS. Also, a traditional evaluate v 
would deliver some personal insights, some 
process information, and come customizing that 
GENESYS cannot— is not designed to-provide. 
On the other hand, GENESYS provides for every 
program a wfde array of statistics, some of which 
were not included in past customized evaluations. 
In general, though, we can estimate the cost of the 
type of traditional evaluation a program would 
have received by calculating the cost of the evalu- 
ation component we would have included in a 
proposal The cost of GENESYS can be more 
precisely estimated based upon the actual re- 
sources that were committed to it during the 
start-up year and in its second year of implemen- 
tation. 

For GENESYS, in the start-up year, the cost was: 



$ 17,000 
40,000 
5,000 
$62,000 

Year two costs were: 

$ 8,500 
25,000 
5,000 
$ 38,500 



Programmer 
Professional Staff Time 
Other Direct Costs 
TOTAL 



Programmer 
Professional Staff Time 
Other Direct Costs 
TOTAL 
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The reader must know that in our school system, 
the data processing costs for evaluation are 
usually not carried in, nor charged directly to, 
evaluations, because ORE is a part of the same 
department as data processing. Therefore, in 
these cost comparisons, the costs for data process- 
ing are not compared; however, our best guess is 
that GENESYS computer demands are less than 
those of a traditional evaluation that would 
require many more individual program runs to 
produce the same output. 

In case the reader wishes to skip the details of 
how we compared costs for traditional evaluations 
and GENES YS, the bottom line numbers will be 
presented first, then the details will follow. 
Figure 2 summarizes the comparisons made. 

Figure 2: Cost Comparisons 





Theoretical Actual Computer 
Traditional GENESYS Lab 
Evaluation Evaluation Evaluation 


Programs wilh a 
GENESYS component 




10 




Programs evaluated 
completely with 
GENESYS 




23 


I 


Total number of 
evaluations 


33 


33 


1 


Average cost per 
evaluation 


$17,600 


$1467 


$4,500 


Total cost for 
33 evaluations 


$577,500 


$38,600 


$4,500 



metacomparisons such as our dropout program 
comparison charts that use GENESYS and other 
data to summarize dropout programs. 

TLe bottom line becomes that a traditional evalu- 
ation would cost about $17,500 to produce the 
GENESYS data, the computer lab evaluation that 
sparked the eventual development of GENESYS 
cost about $4,500, and a current GENESYS 
evaluation costs about $1,167. 

If the District were to fund traditional evaluation 
activities for the 33 programs included in the 
1989-90 GENESYS runs, the cost could have been 
$577,500 instead of the $38,500 actually spent on 
GENESYS. 

In today's real world, the cost of doing a tradi- 
tional evaluation is coming down in the are<*£ of 
data analysis, but still remains high in the labor- 
intensive area of process evaluation. GENESYS is 
not a replacement for first-hand process evalua- 
tion. 

Clearly, a working GENESYS can deliver a large 
number of evaluations for much less than tradi- 
tional evaluations would cost We can, and will, 
debate forever the relative merit of a generic 
evaluation and a customized one. What we cannot 
debate is the fact that GENESYS allows us to get 
a certain level of evaluative information on a large 
number of programs at a very low cost 



The cost for a traditional evaluation was derived 
by examining currently funded programs that 
have an evaluation component and relating them 
to the size and type of program that GENESYS 
has been used to evaluate. By this method, we 
estimated that GENESYS actually replaces 
evaluation resources that cost between $10,000 
and $25,000 in separate budgets. Therefore, we 
used $17,500 as the cost of an average evaluation 
of the type for which GENESYS is appropriate. 
This would also apply to the 10 programs for 
which GENESYS contributes only tf portion of the 
overall evaluation— for example, the Chapter 2 
evaluation that includes much data collection and 
analysis beyond what is provided by GENESYS. 

Not included in these cost estimates is the sav- 
ings/contribution GENESYS makes to 
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ATTACHMENT 1 (Page 1 of 3) 



GENESYS DEFINITIONS— EVALUATION 
SUMMARY 

PROGRAM MEMBERSHIP— DESCRIPTIVE 
INFORMATION 

For each program included in GENESYS, ORE or 
program staff define those to be included (see 
program descriptions)* Most programs or groups 
are for students involved in 1988-89. Some (e.g., 
sixth graders, DARE, and TAP/AIP) are for groups 
served in 1987-88. Descriptive information 
provided for each program includes: 

NUMBER SERVED: Total served (may be 
cumulative, semester, or one point in time 
count). 

ETHNICITY: Percentage Black, Hispanic, and 
Other (includes White, Asian, and Ameri- 
can Indian). 

SEX: Percentage male and female. 

LOW INCOME: Percentage eligible for free or 
reduced-price meals. 

LEP Percentage identified as limited in 
English proficiency (regular or special 
education) and served in bilingual, 
Engtish-as-a-Second Language (ESL), or 
alternative programs as of the end of the 
year (or whenever GENESYS was run). 
Note: Some students "exit" or leave LEP status 
each May once English proficiency is attained. 

OVERAGE FOR GRADE: Percentage older 
than expected for the grade by one or more 
years (as of September I ). Example: 1st 
graders 7 or more on September 1. 

SPECIAL EDUCATION: Percentage of 

students in special education of any type. 

GIFTED/TALENTED: Percentage of students 
in gifted/talented programs. At the 
elementary level, this means participation 
in the AIM High Program. Secondary 
students are counted as gifted if they take 
one or more honors courses. 



OUTCOME INFORMATION 

Outcome information, unless noted, accesses the 
most current data available through VSAM files 
on the computer. Variables include: 

ATTENDANCE: Mean percentage attendance 
(days attended divided by days enrolled) 
for fall and spring of 1989-90 and 1988*89. 
Data for 1988-89 are for those enrolled in 
the 1989-90 program who were active in 
AISD in 1988-89 

DISCIPLINE: Percentage of students involved 
in serious discipline incidents (corporal 
punishment, suspension, expulsion) in fall 
and spring of 1989-90 and 1988-89. 

GRADES: Indicates mean credits earned 
(CREDITS EARNED), number of Fs (#F), 
number of courses with no grade (NO 
GRADE), and grade point average (GPA) 
for high school; indicates grade point 
averages and Ps for junior high/middle 
school. Information is shown for fall and 
spring of 1989-90 and 1988-89. A normal 
course load is five or six classes (2.5 to 3.0 
credits) per semester. The grade point 
average (GPA) is calculated without 
courses in which no grade has yet been 
assigned; it includes Ps and passing 
grades based on a point system of 1-100 
points with 70 as passing. The grade point 
scale for converting numerical scores to 
regular course grade points is included 
below: 



Numerical 


Regular Course Honors Course 


Scores 


Grate Feint 


Grade Poi^ 


97-100 


4.5 


5.0 


93-96 


4.0 


4.5 


80-92 


3.5 


4.0 


87-89 


3.0 


3.5 


83-86 


2.5 


3.0 


80^*2 


2.0 


25 


77-79 


1.5 


2.0 


73-76 


1.0 


1.5 


70-72 


0.5 


1.0 
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ATTACHMENT 1 (Pa_Jof3) 



(Source for grades and credits: SGR History File-SGRH) 
(Source for conversion table: Board Policy Manual, Austin 
ISD» Volume 1) 

DROPOUTS: Percentage of students who 

dropped out of school by the end of the fifth 
six weeks of the 1989-90 school year. The 
percentage who dropped out over the 
entire 1989-90 school year, including the 
summer of 1990, were available in fall, 
1990. 

RETAINED: End oT Year: Percentage of 

students recommended for retention as of 
Ma>, 1990, NOTE: Some students may 
not eventually be retained, especially at 
the secondary level. Successful completion 
of summer school courses or correction of 
grades can result in promotion. Also, at 
the high school level, students rep at only 
courses failed. A "retained" lalx* dimply 
means students have not earned 5, 10, or 
15 credits to be promoted to grades 10, 11, 
and 12, respectively. Also, some special 
education categories are listed as retained 
until schools provide promotion data. 
Beginning of Yean Percentage of students 
actually retained as of the beginning of 
the 1990-91 school year. This figure was 
available in fall, 1990. 

ITBS/TAP: Median percentiles (%iles) of 
group along with number of students 
tasted in Reading Comprehension, Math* 
eraatics Total, and Composite. Composite 
scores include: 

Grades 1-2: ITBS Vocabulary, Reading 
Comprehension, Mathematics Total, 
Spelling, and Word Analysis 

Grades 3-8: ITBS Vocabulary, Reading 
Comprehension, Mathematics Total, 
Language Total, and Work Study Total 



Grades 942: TAP Reading Comprehen- 
sion, Mathematics Total, Written Expres- 
sion, Using Information, Socip! Studies, 
and Science 

TEAMS: Percentage and number of students 
tested who mastered each test — Reading, 
(Language Arts for Exit Level TEAMS, 
Mathematics, and Writing. Mastery levels 
are p<?t yearly by TEA based on a scale 
score of 700 on each test. 

ROSE: The Report on School Effectiveness 
(ROSE) compares Reading Comprehen- 
sion and Mathematics Total grade equiva- 
lent (GE) scores for spring, 1989, and 
spring, 1990, to determine if gains 
achieved are above (♦), below (-), or at (~) 
predicted levels based on regression 
analyses. All students in a grade in a 
program are treated a* a group. ROSE 
predictions for groups with less than 20 
students (*) are not reliable (and are 
therefore not shown). The gain, predicted 
score, and amount over or under the actual 
score compared to the predicted score for 
the group are shown for reference. See 
ORE Publication Letter 89 J for more 
information about the ROSE procedure. 

All A1SD comparison statistics were defined as 
shown above. Students were included if: 

• In grades pre-K through 

• Actively attending a regular campus as of 
February 5, 1990, (The Alternative 
Learning Center and Robbing were 
included for both high school and middle 
school/junior high.) 
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ATTACHMENT 1 (Page S of S) 



GENESYS STATISTICS AND "OFFICIAL* 
AISD COUNTS 

These definitions and inclusion rules vary slightly 
from those used for "official* AISD counts. For 
example, students were included in GENESYS if 
they were active as of midyear (February 5, 1990). 
Published district wide ITBS/TAP median percen- 
tiles will therefore differ from those presented 
here because all test takers were included, 
whether or not they were active in February. 



GENESYS COMPARISONS— EXECUTIVE 
SUMMARY 

Outcome data for each group included in GENE- 
SYS are compared to national and District aver- 
ages to provide a meaningful context for judg- 
ments about program effectiveness. The following 
comparisons are made* 



YftriaMa 

ITBS/TAP Achievement 

TEAMS Achievement 

Attendance 

Discipline 

Grades 

Retainers 
Dropouts 



1988 national norms; 
Predicted achievement 
with actual achievement 

AISD averages in mathematics, 
reading (language arts at 
Exit Level), and writing 

AISD attendance rates 

AISD discipline rates 

Grade point averages (GPA's) 
(secondary only) 
for all AISD students 

AISD retention rates 

AISD dropout rates; 
(grades 7*12 only) 
Predicted rate with obtained 
dropout rate* 



* Implemented in summer, 1990 

On all variables, comparisons are made to the 
appropriate grade or grade span — elementary 
(grades pre-K-6), middle/junior high (grades 6*8), 
and high school (grades 9-12). For example, 
performance on the ITBS by students in grade 3 
in the GENESYS group is compared with the 
national norm for grade 3. The retention rate for 



hign school students in a GENESYS group is 
compared with the retention rate for all AISD 
high school students. 

On most of the above variables, the comparison 
made is to the AISD average or rate, in other 
words, to the general student population (at the 
appropriate grade span). There are two excep- 
tions in which the comparison is not to the general 
population: 

1. By means of ROSE (see Pub. Letter 89.J), 
ITBS/TAP achievement levels for program 
students are compared with predicted 
achievement levels for students with 
similar characteristics. 

2. Beginning in summer, 1990, the dropout 
rate predicted for program students was 
compared with their actual dropout rate. 

Many comparisons to the outcome data for pro- 
gram students could be made. Comparison to the 
general population contrasts the performance of 
the program group with that of students overall. 
This comparison has the advantage of pointing up 
clear differences in performance where the pro- 
gram group is highly select, e.g., honors students. 
On the other hand, comparisons like ROSE, which 
take into account the program students' charac- 
teristics, will continue to be sought so that GENE- 
SYS can become even more usefiil in the future. 
In the meantime, users desiring other comparison 
groups than the general population have the 
option to identify the students and have GENE- 
SYS run on the groups they define. 
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ATTACHMENT 2 



Sample GENESYS Printout for Data by Student 
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ATTACHMENT 3 ( Page 1 of 4) 



GENESYS PROGRAM DESCRIPTION 



PROGRAM NAME: 



Liberal Arts Academy (Johnston) 



EVALUATION CONTACT: 



Vince Faredes 



PROGRAM CONTACT: 



Clark Lyman 



m Funding (Local, State, or Federal): Local 

• Budget Allocation: $449,693 

• Number of campuses with program: 1 - Johnston High School. 
Representatives from all public middle/ junior highs, all 
attendance areas. 

• Eligibility/students served: 

1. ITBS Language and Reading total 

2. GPA - (middle/ junior high) 

3. Most recent grades 

4. Application essay 

5 . Interview 

6. Two or more teacher recommendations 

Staff takes into account all of the above to best place the 
student whether in LAA, Science Academy, or Honors courses. 

m Grades served: 9, 10 (2nd year of program)* Eventually 
9-12 (one grade per year will be added) • 

• Source of file: Roster with all in program as of January 1990. 

m Subject areas taught: 7-period academic day 
Foreign language 
LAA English 
LAA Social Studies 
Science 
Mathematics 
Health/PE 

Selected electives (must be approved) - Band, Drama, 
Journalism, Dance, Debate 

» Program focus/goals/methods: The Liberal Arts Academy at 
Johnston High School provides gifted, creative , and talented 
students an accelerated academic program leading to an excep- 
tionally strong preparation for college* It is expected that 
students will graduate at the end of four years with one year's 
college credit* Capable students and their LAA families are 
interested in general preparation in all liberal arts areas and 
special enrichment in the areas of foreign languages and the 
humanities. Additionally, the Liberal Arts Academy provides 
study trips, resource speakers, and numerous cultural oppor- 
tunities to its student scholars on an ongoing basis. 
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ATTACHMENT 3 ( Page 2 of 4) 



EXECUTIVE SUMMARY 

SENIOR HIGH 
GRACES 9-12 

LIBERAL ARTS ACADEMY AT JOHNSTON, 1989-90 



GROUP CHARACTERISTICS: 

Number of students In this group: 1^5 

Percent low income: 1§ 

Percent minority: 32 

Percent female: ©8 

Percent limited English prof icient (LEP) : I 

Percent overage for their grade: 8 

Percent special education students: t 

Percent gifted/talented students: 97 



Major F i ndi ngs 

TAP ACHIEVEMENT: The spring, 1990, Tests of Achievement and Proficiency (TAP) 
median percentile scores of program students were compared to 
the 1988 national norms. 



Out of k comparisons* program 
students' scores were... 

Read i ng Mathemat i cs 
Above the national norm in 2 2 
At the national norm in 0 0 
Below the national norm in 0 0 



TAP scores from spring, 1990* were compared to predicted levels 

of achievement by means of the Report on School Effectiveness (ROSE) 

procedure. 

Out of k comparisons* program 
students* scores. . . 

Read i ng Ha t hema tics 

Exceeded predicted levels in 1 0 

Achieved predicted levels in 1 2 

Were below predicted levels in 0 0 

Were too few for analysis in 0 0 



TEAKS ACHIEVEMENT: Compared to the A (SO averages in mathematics* 
r^a^tng, and writing, the percentages of program students mastering 
the TEAMS at grades 9 and II (first-time test takers) were: 

Read 1 ng/ Mathematics Wr i 1 1 ng 

Language Arts 

Higher in 1 11 

The same in 0 0 0 

Lower in 0 0 0 



ATTENDANCE: Compared with the attendance rates for senior high 
distr ictwide: 

The program A I SO Program 

rate was • * , 

Fall, 1989 Higher 92.6% 37. }X 

Spring* 1990 Higher §0.8* §5-°* 

Compered to*.. I989~90 program attendance was... 

Program students Fall Higher 

in T988-89 SprLv: Higher 
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ATTACHMENT 3 ( Page 3 of 4) 



DISCIPLINE: CofflM~ed with the percentages of students involved in 
discipline incidents at the senior high level di sir ictwide: 

Program 



fall. 1989 
Spring, 1990 

Compared to. . . 



Proor 

in 



Students 
9 



The program 
rate was. . . 

Lower 

Lower 



AlSD 



O.O* 
0.7% 



1989*90 program discipline was 

fall: Lower 
Spr ing: Higher 



GRADES: Compared with the GPA 1 s for all AlSD senior high students: 

AlSD Program 



Fall, 1989 
Spring, I 990 

Compared to. . 



Progn 
in 



students 
'89 



The program 
rate was. . . 

Hi gher 

Higher 



8:8 8:8 

1989~90 program GPA was... 



Fall: Lower 
Spr ing: Lower 



RETAiNEES/DRQPOUTS: Comparing the percentage of program students 
recommended in spring, »990# for retention the following year with 
all AlSD senior nigh students: 



The program 
rate was. . . 
Lower 



A I r 3 Program 
16. i>* k.Bt 



Compared to the fifth six weeks dropout rate for senior high students 
for 1989-90: 



The program 
rate was. . * 
Lower 



A I SO Program 
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89.30 

GENESYS 

GENaoc EwtfufftO" SYStftm 

PROGRAM/GROUP: liberal arts academy at ,i0hnST0n, 



AUSTIN INDEPENDENT SCHOOt DISTRICT 
DEPARTMENT OF MANAGEMENT INFORMATION 
OFFICE OF RESEARCH AND EVALUATION 



EVALUATION 
SI M MARY 



1 989 -90 



PRINT DATE: o?/io.-90 



DEMOGRAPHIC INDICATORS 



Grade 

• Students 



PK 



8 



9 

?9 



10 

66 



11 



Male F«ul« 

4? 98 
32 68 



Ethnicity Low 
Black Hispanic Other 

20 34 9* 28 

14 23 63 19 



LEP 

1 



Overage 
For Grade 

!2 

a 



Special 
Education 

i 
i 



PRQGRESjyNOICATORS 



DropOUta : Q Q* EVP QF THE STH 6 WEEKS ROtotnOQ) 

CTMItl 



89-90 * 
% 

88-89 * 
% 



Attendance 
Fall Spring 

14$ 142 

97 3 95 6 

132 134 

93 T 95 2 



OttClpI If 
Fall Spring? 
O i * 

oo of ;avo 

2 O j* 

14 0 0 AVQ 



Fall Spring 

144 14D 

* 3 3 2 

63 63 

3 2 3 3 



End of Year 

e>™0 
Fall Spring 

144 t4Q 

O 26 0.31 

63 63 

0 30 0 22 



Grade 

Reading Coop r anon t 1 on 
NMar of Students 
Tielheejatlei Total 
tr of Students 



ACHIEVEMENT INDICATORS 



ITBS/TAP MEDIAN PERCENTILES, 1989-90 
1 2 3 4 B 8 



ISoepbette ~ 

»r oF Students 



Or ado 

READING COMPREHENSION 

Nuaoar of studonti 
1989 Grade Equivalent 
t990 Grade Equivalent 
Oat ft 

Predicted Score 
Over/ Under Actual 
Stgntf tcance 

MATHEMATICS TOTAL 

Nuaber of Students 

1989 Grade Equivalent 

1990 Grade Equivalent 
Qain 

Predicted Score 
Over /under Actual 
Significance 



9 

84 

77 

"?3 
77 

~8 1 
73 



10 

86 
62 
^8 
62 
83 
6* 



ROSE. SPRING 1989 
2 3 4 



TO SPRING 
9 6 



1990 
7 



MEAN G'iADt EQUIVALENT 
6 8 10 11 



9 


10 


63 


54 


lO 6 


14 0 


14 4 


15 7 


3 8 


i 6 


13 4 


«5 i 


i 0 


OS 


♦ 




63 


%b 


9 5 


*3 5 


12 6 


14 3 


3 O 


O 9 


«2 2 


<4 5 


0 4 


- 1 


* 





12 TOTAL 

145 

Oif ted/ 
Talented 

14 1 
9" 



4_ a % _ j nn 1 ng of veer 
*No Grade* 
Fall Spring 

144 140 

O OS C 16 

63 63 

0 00 0 06 



CPA 
Fall Spring 

144 140 

86 7 95 7 

63 63 

85 1 86 4 



It 



12 



12 



TEAMS PERCENT MASTERING 
Grade 3 8 



Mathemttca 

»r of Student* 



Reading/ Language Arts 
Nuaber of Students 



Nrftfng 



of Students 



8 

97 
78 
10O 

.. 7 8 
97 
75 



11 



ERIC 



18 



21 
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GENESYS 
"BEFORE" PROCESSING 



START 



Select programs 



f 



Decide 
wnrcn students 



ODta»n 
student t D's 
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No 
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it necessary 
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Ret i ne f * i o 
i f necessary 



f Student tD s 
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1 " 
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r -' 
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GENESYS 

"AFTER" PROCESSING 





Dtssemineto 




results to 


fr> 






evaluation and 




program staff 



Explain results/ 
answer questions 



Modify input fiie(s) 
and return 




input 
ft ie(s) 
cor rect ** 



Process feedback 




and consider 




future cnanges 


■ ■ - - ■■■ ■ ■ 



1 
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i" ' 
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ATTACHMENT 5 



Requirements for GENES YS Data Files 

• Data files should contain the student ID 
numbers of the students in the group. 

• There should be one ID per line beginning in 
column 1. There is no limit on the number of 
students who may be in a group, but because 
of the computer running time that GENESYS 
requires, groups must contain a minimum of 
25 students. 

• Groups must be defined as either elementary, 
middle/junior high school, or high school, and 
each file must contain the ID numbers only for 
students within one of these divisions. If you 
have a group whose grade levels span these 
divisions, you will need to separate the group 
into the appropriate grade spans; i.e., ycu will 
need separate files. For example, if you have 
a group with students in grades 7-12, you will 
need to create two files, one with the ID's for 
students in grades 7-8, and a second with the 
ID's for students in grades 9-12, 

• The ID's on data files should be checked to 
eliminate bad ID's and duplicate ID's. Veda 
has written a program to use for this purpose: 
DWSCMPAR (ORWSAS). 

• Data files should be given eight-character 
names beginning with GE@, e.g., GE@GRADH 
for high school students served by Project 
GRAD. Data files should be placed in 
ORSSAS. 

• Give your group/program a name not to 
exceed 52 characters. This name will appear 
as a title on the Executive Summary and on 
the Evaluation Summary. Try to include the 
full name of the program rather than an 
abbreviation, and include the year, e.g., 
TEACH AND REACH, 1989-90. If you are 
following a group that was constituted prior to 
this year, use a title which makes clear which 
year refers to the group and which is the year 
the analysis was done, e.g., SPR '89 TRANSI- 
TIONAL ACADEMIC PROGRAM, IN 1980- 
90. 



• Specify which grade levels the students in 
your group/program are in. The grade levels 
you indicate will appear as a second title 
under the name of the program on the Execu- 
tive Summary. For the sake of clarity, do not 
indicate a whole grade span if students are 
only in one grade. For example, only students 
in grade 9 are served in the Transitional 
Academic Program. The title should read 
GRADE 9, rather than GRADES 9-12. 

Types of Data Files 

The GENESYS file sheet lists three different 
types of data files: 

• Cumulative, 

• Point in time, and 

• Point in time with service conditions. 

On a cumulative file, every student served by the 
program at any time during the year, whether the 
student is currently served, is currently inactive, 
or even has left the program or the District, is 
entered. 

The point-in-time file includes all the students 
being served at a particular point in time, without 
regard for students who were formerly served or 
for the length of service to students at the ume 
the file is built or in the future, 

The point-in-time with service conditions file 
contains students served at a particular point in 
time but places conditions on which students are 
included based, for example, on the students' 
length of service. It may be desirable, under this 
condition, to "capture* on the file only those 
students who have received services for at least 
some minimum length of time — arguably the most 
"stable" students or the students on whom the 
program's intervention has had a chance to take 
effect Besides length of service, another condition 
which might be imposed is that students be active 
on the Student Master File. 

It does not matter to GENESYS what sort of file 
you have, in terms of its processing, but the 
distinction needs to be taken into account in 
interpreting the information GENESYS produces. 
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ATTACHMENTS 



Ideas for GENES YS Enhancements 

• Program summary charts similar to the data- 
by -student report. These charts would com- 
pare statistics across multiple programs 
selected by the user, Districtwide summaries, 
by grade span, would 1% included among the 
programs. 

• Numbers and percentages of students for all 
variables. Only percentages of retainees and 
dropouts are presently reported. 

• More "user-friendly* programming, and brief 
training for other programmers, so that other 
programmers and noncomputer programmers 
can submit their own runs. 

• Methods for overcoming slowdowns caused by: 
Deciding who should be included in data files, 
Deciding what sources should be used for files, 
and 

Difficulty in collecting basic program informa- 
tion. 

• A comparison of expected and obtained 
dropout rates for junior high school and high 
school programs. 

• Additional cross-tabulations of variables (e.g., 
grade by ethnicity, etc,) available on request. 

• For programs where students may earn 
eighth- and ninth-grade credits, an evaluation 
summary showing middle/junior high school 
and high school credits on the same sheet or 
on separate sheets with appropriate labels. 

• A staff summary sheet (similar to that in the 
Annual Performance Report). 

• A budget summary based on budget codes 
(similar to the District's budget book). 

• Significance tests with probability levels 
between groups and between pre- and posttest 
measures printed. 

• Executive summaries with comparisons made 
between groups in addition to the present 
comparisons between a single group and 
District totals. 
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